IIoT asymmetric reactive buffering to dynamically manage and optimize in-motion data batch size

ABSTRACT

A reactive buffering system for use in IIoT data pipelines dynamically adjusts data accumulation and delivery by a node of a pipeline based on aggregated downstream metrics representing current data processing latencies of downstream nodes. Based on these downstream performance metrics, a reactive node that adjusts the size of the next data batch to be sent to an adjacent downstream node. The nodes of the data pipeline are configured to support a request-response based handshaking protocol whereby the nodes that send data to downstream nodes maintain up-to-date performance level information from adjacent downstream nodes. With this performance information, together with pipeline priorities, the sending node (or reactive node) adjusts the transmission rate and intermediate buffering of data. In this way, the nodes of the pipeline can dynamically regulate interim data storage to avoid overwhelming the pipeline system with too much data during periods of high latency.

BACKGROUND

The subject matter disclosed herein relates generally to industrialautomation, and, more particularly, collection of industrial data.

BRIEF DESCRIPTION

The following presents a simplified summary in order to provide a basicunderstanding of some aspects described herein. This summary is not anextensive overview nor is intended to identify key/critical elements orto delineate the scope of the various aspects described herein. Its solepurpose is to present some concepts in a simplified form as a prelude tothe more detailed description that is presented later.

In one or more embodiments, a node system is provided, comprising a datainput component configured to receive industrial data from an adjacentupstream node system of a data pipeline and to queue the industrial datato yield queued data; a data output component configured to send subsetsof the queued data to an adjacent downstream node system of the datapipeline in data batches; a batch size calculation component configuredto determine a size of a next data batch, of the data batches, based onperformance metric data received from the adjacent downstream nodesystem; and a batch size setting component configured to instruct thedata output component to send the next data batch in accordance with thesize determined by the batch size calculation component.

Also, one or more embodiments provide a method, comprising queuing, by anode system comprising a processor, industrial data received from anadjacent upstream node system of a data pipeline to yield queued data;determining, by the node system, a size of a next data batch to be sentto an adjacent downstream node system of the data pipeline based onperformance metric data received from the adjacent downstream nodesystem; and sending, by the node system, a subset of the queued data tothe adjacent downstream node system of the data pipeline as the nextdata batch in accordance with the size.

Also, according to one or more embodiments, a non-transitorycomputer-readable medium is provided having stored thereon instructionsthat, in response to execution, cause a node system comprising aprocessor and acting as a node of a data pipeline to perform operations,the operations comprising queuing industrial data received from anadjacent upstream node system of the data pipeline to yield queued data;determining a size of a next data batch to be sent to an adjacentdownstream node system of the data pipeline based on performance metricdata received from the adjacent downstream node system; and sending asubset of the queued data to the adjacent downstream node system of thedata pipeline as the next data batch in accordance with the size.

To the accomplishment of the foregoing and related ends, certainillustrative aspects are described herein in connection with thefollowing description and the annexed drawings. These aspects areindicative of various ways which can be practiced, all of which areintended to be covered herein. Other advantages and novel features maybecome apparent from the following detailed description when consideredin conjunction with the drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a diagram illustrating an example IIoT data pipeline.

FIG. 2 illustrates a high-level overview of an architecture in whichdata from an industrial enterprise can be migrated to a cloud platform.

FIG. 3 is a block diagram of an example pipeline node system.

FIG. 4 is a diagram illustrating data processing latencies involved inmoving data across nodes of a IIoT data pipeline.

FIG. 5 is a diagram illustrating a network of node systems that make upa portion of one or more IIoT data pipelines.

FIG. 6 is a diagram illustrating the first two steps of an example datatransmission cycle that can be carried out by one or more embodiments ofthe data pipeline node systems.

FIG. 7 is a diagram illustrating subsequent three steps of the exampledata transmission cycle that can be carried out by one or moreembodiments of the data pipeline node systems.

FIG. 8 is a diagram illustrating calculation of the cumulativeprocessing latency.

FIG. 9 is a graph representing an example relationship betweendownstream performance latency relative to a reactive node and a databatch size that is to be set for the reactive node as a function of thedownstream latency.

FIG. 10 is a diagram illustrating calculation of a batch size by a batchsize calculation block.

FIG. 11 is a diagram illustrating adjustment of a reactive node's batchsize based on results of a batch size calculation.

FIG. 12 is a diagram illustrating monitoring of timestamp data by atimestamp moderator agent.

FIG. 13 is a diagram illustrating delivery of layer-specific timereferences to node systems that make up the data pipelines.

FIG. 14 is a block diagram of an example timestamp moderator agent.

FIG. 15 a is a diagram illustrating delivery of a time reference tonodes of a layer of a parallel data pipeline network.

FIG. 15 b is a diagram illustrating alignment of timestamped data withina layer of a parallel data pipeline network by regulating delivery ofdata batches by nodes of the layer.

FIG. 16 is a flowchart of an example methodology for regulating a sizeof a data batch sent by a reactive node of an industrial IIoT datapipeline to an adjacent downstream reactive node.

FIG. 17 is a flowchart of an example methodology for both regulating asize of a data batch sent by a reactive node of a parallel data pipelinenetwork to an adjacent downstream reactive node, as well as maintainingsynchronization of timestamped data within respective node layers of theparallel data pipeline network.

FIG. 18 is a flowchart of an example methodology for maintainingsynchronization of data stamps within a node layer of network ofparallel IIoT data pipelines.

FIG. 19 is a flowchart of an example methodology for generating areference timestamp to be used by nodes of a layer of a parallel IIoTdata pipeline network to maintain synchronization of timestamps withinthe nodes.

FIG. 20 is an example computing environment.

FIG. 21 is an example networking environment.

DETAILED DESCRIPTION

The subject disclosure is now described with reference to the drawings,wherein like reference numerals are used to refer to like elementsthroughout. In the following description, for purposes of explanation,numerous specific details are set forth in order to provide a thoroughunderstanding thereof. It may be evident, however, that the subjectdisclosure can be practiced without these specific details. In otherinstances, well-known structures and devices are shown in block diagramform in order to facilitate a description thereof.

As used in this application, the terms “component,” “system,”“platform,” “layer,” “controller,” “terminal,” “station,” “node,”“interface” are intended to refer to a computer-related entity or anentity related to, or that is part of, an operational apparatus with oneor more specific functionalities, wherein such entities can be eitherhardware, a combination of hardware and software, software, or softwarein execution. For example, a component can be, but is not limited tobeing, a process running on a processor, a processor, a hard disk drive,multiple storage drives (of optical or magnetic storage medium)including affixed (e.g., screwed or bolted) or removable affixedsolid-state storage drives; an object; an executable; a thread ofexecution; a computer-executable program, and/or a computer. By way ofillustration, both an application running on a server and the server canbe a component. One or more components can reside within a processand/or thread of execution, and a component can be localized on onecomputer and/or distributed between two or more computers. Also,components as described herein can execute from various computerreadable storage media having various data structures stored thereon.The components may communicate via local and/or remote processes such asin accordance with a signal having one or more data packets (e.g., datafrom one component interacting with another component in a local system,distributed system, and/or across a network such as the Internet withother systems via the signal). As another example, a component can be anapparatus with specific functionality provided by mechanical partsoperated by electric or electronic circuitry which is operated by asoftware or a firmware application executed by a processor, wherein theprocessor can be internal or external to the apparatus and executes atleast a part of the software or firmware application. As yet anotherexample, a component can be an apparatus that provides specificfunctionality through electronic components without mechanical parts,the electronic components can include a processor therein to executesoftware or firmware that provides at least in part the functionality ofthe electronic components. As further yet another example, interface(s)can include input/output (I/O) components as well as associatedprocessor, application, or Application Programming Interface (API)components. While the foregoing examples are directed to aspects of acomponent, the exemplified aspects or features also apply to a system,platform, interface, layer, controller, terminal, and the like.

As used herein, the terms “to infer” and “inference” refer generally tothe process of reasoning about or inferring states of the system,environment, and/or user from a set of observations as captured viaevents and/or data. Inference can be employed to identify a specificcontext or action, or can generate a probability distribution overstates, for example. The inference can be probabilistic—that is, thecomputation of a probability distribution over states of interest basedon a consideration of data and events. Inference can also refer totechniques employed for composing higher-level events from a set ofevents and/or data. Such inference results in the construction of newevents or actions from a set of observed events and/or stored eventdata, whether or not the events are correlated in close temporalproximity, and whether the events and data come from one or severalevent and data sources.

In addition, the term “or” is intended to mean an inclusive “or” ratherthan an exclusive “or.” That is, unless specified otherwise, or clearfrom the context, the phrase “X employs A or B” is intended to mean anyof the natural inclusive permutations. That is, the phrase “X employs Aor B” is satisfied by any of the following instances: X employs A; Xemploys B; or X employs both A and B. In addition, the articles “a” and“an” as used in this application and the appended claims shouldgenerally be construed to mean “one or more” unless specified otherwiseor clear from the context to be directed to a singular form.

Furthermore, the term “set” as employed herein excludes the empty set;e.g., the set with no elements therein. Thus, a “set” in the subjectdisclosure includes one or more elements or entities. As anillustration, a set of controllers includes one or more controllers; aset of data resources includes one or more data resources; etc.Likewise, the term “group” as utilized herein refers to a collection ofone or more entities; e.g., a group of nodes refers to one or morenodes.

Various aspects or features will be presented in terms of systems thatmay include a number of devices, components, modules, and the like. Itis to be understood and appreciated that the various systems may includeadditional devices, components, modules, etc. and/or may not include allof the devices, components, modules etc. discussed in connection withthe figures. A combination of these approaches also can be used.

Industrial controllers and their associated I/O devices are central tothe operation of modern automation systems. These controllers interactwith field devices on the plant floor to control automated processesrelating to such objectives as product manufacture, material handling,batch processing, supervisory control, and other such applications.Industrial controllers store and execute user-defined control programsto effect decision-making in connection with the controlled process.Such programs can include, but are not limited to, ladder logic,sequential function charts, function block diagrams, structured text, orother such programming structures.

Because of the large number of system variables that must be monitoredand controlled in near real-time, industrial automation systems oftengenerate vast amounts of near real-time data. In addition to productionstatistics, data relating to machine health, alarm statuses, operatorfeedback, electrical or mechanical load over time, and the like areoften monitored, and in some cases recorded, on a continuous basis. Thisdata is generated by the many industrial devices that make up a typicalautomation system, including the industrial controller and itsassociated I/O, telemetry devices for near real-time metering, motioncontrol devices (e.g., drives for controlling the motors that make up amotion system), visualization applications, lot traceability systems(e.g., barcode tracking), etc. Moreover, since many industrialfacilities operate on a 24-hour basis, their associated automationsystems can generate a vast amount of potentially useful data at highrates. The amount of generated automation data further increases asadditional plant facilities are added to an industrial enterprise.

To gain insights into the operation of plant-floor automation systemsand processes, this high-density industrial data can be collected andstreamed to an analytics, visualization, or reporting system residing ona cloud platform or another high-level platform via a data pipeline, ora network of parallel data pipelines, as part of an industrial internetof things (IIoT) architecture. FIG. 1 is a diagram illustrating anexample IIoT data pipeline. Data pipeline 114 can comprise a series ofchained nodes 104 capable of relaying aggregated industrial data 102from an edge device 106 residing on the plant floor to cloud storage110. Nodes 104 may be server devices, microservices executing onrespective computer hardware platforms, or other such processingelements. Each of the chained nodes 104 receives data from an adjacentupstream node 104 and passes this data to an adjacent downstream node104 to thereby convey data from the source to the destination. In somearchitectures, a sending node can be connected to multiple upstreamfeeder nodes and/or multiple downstream receiving nodes. In addition toconveying collected industrial data through the pipeline 114, a givennode 104 may also perform processing on received data in accordance witha data processing application installed on the node 104. Suchapplication may include, but are not limited to, notificationapplications that generate and send notifications to specified clientdevices if any of the data satisfies a defined notification criterion,data transformation applications that transform or reformat the data tosuit the needs of a target application (e.g., filtering, data reduction,anomaly detection, aggregation, compression, truncation,contextualization), or other such applications.

Any of the nodes 104 or edge device 106 may perform processing on thecollected data 102 as the data is streaming through the pipeline 114,and as such the data 108 that is delivered to the cloud storage 110 maybe a processed or reduced version of the data 102 collected from theplant floor devices and sensors that make up the plant floor automationsystems. Once the data 108 has been moved to cloud storage 110, the data108 can be analyzed or visualized by high-level applications 112.

In some architectures, the edge device 106 may collect data from theindustrial devices at fixed data collection rates, while the nodes 104may process and convey incoming data at variable throughputs. IIoTsolutions that include parallel data pipelines (e.g., chained nodes 104)typically lack prioritization strategies for delivering industrial data,and do not regulate the volume of data transfer between nodes of thepipeline 114. In industrial applications in which large volumes of dataare being streamed from the plant floor to the cloud platform, this lackof prioritization and regulation can result in data congestion orchoking of the pipeline 114, particularly when nodes 104 fail to adjustto spikes in upstream data volume or when downstream nodes 104experience slow data throughput. This data congestion can negativelyimpact the pipeline's data delivery reliability. Moreover, when datacongestion is experienced at a node 104 of the pipeline 114, thiscongestion can quickly propagate to upstream nodes 104, and if notaddressed may lead to critical data loss or long disaster recoverycycles.

To address these and other issues, one or more embodiments describedherein provide a reactive buffering system for use in IIoT datapipelines. The reactive buffering system can dynamically adjust dataaccumulation and delivery by a node of a pipeline based on aggregateddownstream metrics (ADM) representing current data processing latenciesof downstream nodes. Based on these downstream performance metrics, thereactive buffering system adjusts the size of the next data batch to besent to the adjacent downstream reduction node. To this end, the nodesof the data pipeline are configured to support a request-response basedhandshaking protocol whereby the nodes that send data to downstreamnodes maintain up-to-date performance level information from adjacentdownstream nodes. With this performance information, together withpipeline priorities, the sending node (or reactive node) can adjust thetransmission rate and intermediate buffering of data. In this way, thenodes of the pipeline can dynamically regulate interim data storage toavoid overwhelming the pipeline system with too much data during periodsof high latency (e.g., busy periods, data spikes, processing issues,etc.).

As noted above, the reactive buffering system described herein can beused as part of an IIoT data pipeline used to migrate data generated atone or more plant facilities to a cloud environment for storage,analysis, reporting, or visualization. FIG. 2 illustrates a high-leveloverview of an architecture in which data from an industrial enterprisecan be migrated to a cloud platform. This architecture is an examplecontext in which embodiments of the reactive buffering system can beused. The enterprise comprises one or more industrial facilities 204,each having a number of industrial devices 208 and 210 in use. Theindustrial devices 208 and 210 can make up one or more automationsystems operating within the respective facilities 204. Exampleautomation systems can include, but are not limited to, batch controlsystems (e.g., mixing systems), continuous control systems (e.g., PIDcontrol systems), or discrete control systems. Industrial devices 208and 210 can include such devices as industrial controllers (e.g.,programmable logic controllers or other types of programmable automationcontrollers); field devices such as sensors and meters; motor drives;operator interfaces (e.g., human-machine interfaces, industrialmonitors, graphic terminals, message displays, etc.); industrial robots,barcode markers and readers; vision system devices (e.g., visioncameras); safety relays, optical safety systems, or other suchindustrial devices.

Industrial automation systems can include one or more industrialcontrollers that facilitate monitoring and control of their respectiveprocesses. These industrial controllers exchange data with the fielddevices using native hardwired I/O or via a plant network such asEthernet/IP, Data Highway Plus, ControlNet, Devicenet, or the like. Agiven controller typically receives any combination of digital or analogsignals from the field devices indicating a current state of the devicesand their associated processes (e.g., temperature, position, partpresence or absence, fluid level, etc.), and executes a user-definedcontrol program that performs automated decision-making for thecontrolled processes based on the received signals. The controller thenoutputs appropriate digital and/or analog control signaling to the fielddevices in accordance with the decisions made by the control program.These outputs can include device actuation signals, temperature orposition control signals, operational commands to a machining ormaterial handling robot, mixer control signals, motion control signals,and the like. The control program can comprise any suitable type of codeused to process input signals read into the controller and to controloutput signals generated by the controller, including but not limited toladder logic, sequential function charts, function block diagrams,structured text, or other such platforms.

Although the example architecture illustrated in FIG. 2 depicts theindustrial devices 208 and 210 as residing in fixed-location industrialfacilities 204, the industrial devices 208 and 210 may also be part of amobile control application, such as a system contained in a truck orother service vehicle.

On-premise edge devices 106 can collect data from industrial devices 208and 210—or from other data sources, including but not limited to datahistorians, business-level systems, etc.—and feed this data into a datapipeline (e.g., pipeline 114 in FIG. 1 ) which migrates the data to thecloud platform 202 for processing and storage. Cloud platform 202 can beany infrastructure that allows cloud services 212 to be accessed andutilized by cloud-capable devices. Cloud platform 202 can be a publiccloud accessible via the Internet by devices having Internetconnectivity and appropriate authorizations to utilize the services 212.In some scenarios, cloud platform 202 can be provided by a cloudprovider as a platform-as-a-service (PaaS), and the services 212 (e.g.,data analysis, visualization, reporting, etc.) can reside and execute onthe cloud platform 202 as a cloud-based service. In some suchconfigurations, access to the cloud platform 202 and the services 212can be provided to customers as a subscription service by an owner ofthe services 212. Alternatively, cloud platform 202 can be a private orsemi-private cloud operated internally by the enterprise, or a shared orcorporate cloud environment. An exemplary private cloud can comprise aset of servers hosting the cloud services 212 and residing on acorporate network protected by a firewall.

Cloud services 212 can include, but are not limited to, data storage,data analysis, control applications (e.g., applications that cangenerate and deliver control instructions to industrial devices 208 and210 based on analysis of real-time system data or other factors),automation system or process visualization applications (e.g., acloud-based HMI), reporting applications, Enterprise Resource Planning(ERP) applications, notification services, or other such applications.Cloud platform 202 may also include one or more object models tofacilitate data ingestion and processing in the cloud.

Ingestion of industrial device data in the cloud platform 202 can offera number of advantages particular to industrial automation. For one,cloud-based storage offered by the cloud platform 202 can be easilyscaled to accommodate the large quantities of data generated daily by anindustrial enterprise. Moreover, multiple industrial facilities atdifferent geographical locations can migrate their respective automationdata to the cloud for aggregation, collation, collective analysis,visualization, and enterprise-level reporting without the need toestablish a private network between the facilities. In another exampleapplication, cloud-based diagnostic applications can monitor the healthof respective automation systems or their associated industrial devicesacross an entire plant, or across multiple industrial facilities thatmake up an enterprise. Cloud-based lot control applications can be usedto track a unit of product through its stages of production and collectproduction data for each unit as it passes through each stage (e.g.,barcode identifier, production statistics for each stage of production,quality test data, abnormal flags, etc.). Moreover, cloud-based controlapplications can perform remote decision-making for a controlledindustrial system based on data collected in the cloud from theindustrial system, and issue control commands to the system. Theseindustrial cloud-computing applications are only intended to beexemplary, and the systems and methods described herein are not limitedto these particular applications. The cloud platform 202 can allowsoftware vendors to provide software as a service, removing the burdenof software maintenance, upgrading, and backup from their customers.

FIG. 3 is a block diagram of an example pipeline node system 302 (alsoreferred to herein simply as a node) according to one or moreembodiments of this disclosure. Aspects of the systems, apparatuses, orprocesses explained in this disclosure can constitute machine-executablecomponents embodied within machine(s), e.g., embodied in one or morecomputer-readable mediums (or media) associated with one or moremachines. Such components, when executed by one or more machines, e.g.,computer(s), computing device(s), automation device(s), virtualmachine(s), etc., can cause the machine(s) to perform the operationsdescribed.

Node system 302, which can be a node of an IIoT data pipeline having atleast some of the functions of nodes 104 described above, can include adata input component 304, a data output component 306, a performancemonitoring component 308, a batch size calculation component 310, abatch size setting component 312, a timestamp alignment component 314,one or more processors 318, and memory 320. In various embodiments, oneor more of the data input component 304, data output component 306,performance monitoring component 308, batch size calculation component310, batch size setting component 312, timestamp alignment component314, the one or more processors 318, and memory 320 can be electricallyand/or communicatively coupled to one another to perform one or more ofthe functions of the node system 302. In some embodiments, components304, 306, 308, 310, 312, and 314 can comprise software instructionsstored on memory 320 and executed by processor(s) 318. Node system 302may also interact with other hardware and/or software components notdepicted in FIG. 3 . For example, processor(s) 318 may interact with oneor more external user interface devices, such as a keyboard, a mouse, adisplay monitor, a touchscreen, or other such interface devices.

Data input component 304 can be configured to receive batches of datafrom an adjacent upstream node system of the data pipeline or from anedge device 106 (if the node system 302 is the first node of a datapipeline). Data output component 306 can be configured to send thereceived data to an adjacent downstream node system of the data pipelinein data batches. As will be described in more detail herein, the nodesystem 302 can adjust the size of the data batch based on downstreamprocessing performance metrics in order to alleviate data congestion inthe event of excessive processing latencies.

Performance monitoring component 308 can be configured to monitor thenode system's aggregated processing performance metrics and to reportthese metrics to an adjacent upstream processing node system. Theseprocessing metrics convey the node system's current processing latenciesso that the upstream node can set the size its next outgoing data batchbased on the node system's current processing capacity. In someembodiments, the performance monitoring component 308 can send theperformance metric data via the same data pipeline channel as that usedto received data at the data input component 304. Alternatively, theperformance monitoring component 308 may send this performance metricsvia a separate dedicated channel.

The batch size calculation component 310 can be configured to calculatea size for the next data batch to be sent to an adjacent downstream nodebased on processing performance metric data received by the downstreamnode. The batch size setting component 312 can be configured to instructthe data output component 306 to output the next batch of data inaccordance with the batch size calculated by the batch size calculationcomponent 310.

The timestamp alignment component 314 can be configured to control atiming of release of the next data batch based on a comparison of thetimestamps associated with the oldest data currently queued on the nodesystem 302 with a timestamp reference generated by a timestamp moderatoragent.

The one or more processors 318 can perform one or more of the functionsdescribed herein with reference to the systems and/or methods disclosed.Memory 320 can be a computer-readable storage medium storingcomputer-executable instructions and/or information for performing thefunctions described herein with reference to the systems and/or methodsdisclosed.

FIG. 4 is a diagram illustrating data processing latencies involved inmoving data across nodes of an IIoT data pipeline. In this example,pipeline node systems 302 ₁₁ and 302 ₂₂ are two adjacent nodes of anexample IIoT data pipeline. When data is being sent from a node (i.e.,node system 302) of a data pipeline to an adjacent downstream node, thenode that is sending the data—e.g., node system S₁₁ in FIG. 4 —isreferred to as the reactive node, while the node that receives and actson the data—e.g., node system S₂₂—is referred to as the reduction node.Since nodes that are not at the extreme ends of the pipeline will bothreceive data from adjacent upstream nodes and send data to adjacentdownstream nodes, these nodes act as both reactive and reduction nodesat various stages of their operation within a given data transmissioncycle. FIG. 4 also depicts a target node 406, which is adjacent to anddownstream from the reduction node S₂₂. Target node 406 may be eitheranother node system 302 of the pipeline, or a final storage destinationfor the data (e.g., a cloud-based storage node). As noted above, nodesystems 302 may be servers, microservices, or other processing elementsthat form the backbone of the IIoT data pipeline.

In-motion data moves from node to node in micro-batches, or databatches. Typically, the size of these micro-batches is fixed. However,to address the data congestion issues discussed above, the reactive nodesystem 302 ₁₁ (S₁₁) described herein is capable of dynamically adjustingthe size of its outgoing data batches 402 based on the processinglatency experienced at the adjacent reduction node system 302 ₂₂ (S₂₂).The processing latency at the reduction node system 302 ₂₂ can becharacterized by a set of time variables that quantify the timesrequired to perform various tasks associated with processing and movingthe data.

For example, T_(rac) represents the time required for the reactive nodesystem 302 ₁₁ to collect the data that is to be included in the databatch 402 from an adjacent upstream data source (e.g., an upstreamreactive node, one or more IIoT industrial devices, or an edge device106). In general, this time to collect the data can be given as theproduct of the size N_(S) of the incoming data batch and the time T_(u)required for the reactive node system 302 ₁₁ to collect a record for asingle data tag of the batch:T _(rac) =N _(s) *T _(u)  (1)

The time required for the reduction node system 302 ₂₂ to read the databatch 402 from the reactive node system 302 ₁₁ is given by T_(rdr). Uponreading the data batch 402, the reduction node system 302 ₂₂ may applyprocessing to transform the data contained in the data batch 402. Insome applications this transformation may include applying datareduction processing to the data in order to reduce the data set; e.g.,by detecting and removing anomalous data or outlier data, or by applyinganother type of filtering algorithm. In some scenarios, the reductionnode system 302 ₂₂ may process the incoming data batch 402 according toa node-specific application that executes on the node system (e.g., anotification application, an analytic application, etc.). The timerequired for the reduction node system 302 ₂₂ to reduce, process, orotherwise transform the data is given by T_(rdt). The time required forthe reduction node system 302 ₂₂ to then write the resulting reduceddata batch 404 to the target node 406 is represented by T_(rdw). Thetotal time T_(rd) required for the reduction node system 302 ₂₂ toreceive, process, and send the incoming data batch 402 can thus be givenasT _(rd) =T _(rdr) +T _(rdt) +T _(rdw)  (2)

The performance metrics for the reactive node system 302 ₂₂ at a giventime can be represented, in part, in terms of the latencies representedby equation (2), which represent the latencies associated withprocessing tasks performed by the reactive node system 302 ₂₂ itself inconnection with receiving, processing, and sending the data. In general,the latency experienced at a given node system 302 is a function of thenode's compute power, the network bandwidth, the complexity of theapplication applied to the data by the node, and the volume of datapassing through node. Additionally, the total latency experienced by agiven reactive node system 302 ₂₂ may also be a function of thelatencies experienced by any adjacent downstream nodes. That is, sincethe data processing latency of a downstream node can propagate toupstream nodes in a cumulative manner, the cumulative latency associatedwith a given reactive node system 302 ₂₂ is a function of both theinherent processing latencies of the node as well as downstreamlatencies that impact the ability of the reactive node system 302 ₂₂ tosend its data batch to the next downstream node.

FIG. 5 is a diagram illustrating a network of node systems 302 that makeup a portion of a parallel data pipeline architecture. Although the flowof industrial data through these data pipelines is in the left-to-right(downstream) direction in FIG. 5 , embodiments of the node systems 302described herein can report their measured cumulative performancemetrics M_(ij) (as measured by the performance monitoring component 308of each node system 302) to adjacent upstream node systems 302 byback-propagating this performance metric information in theright-to-left (upstream) direction. That is, in addition to conveyingdata batches 402 in the downstream direction toward the destination node(e.g., cloud storage or another destination), node systems 302 can alsosend performance metric data 502 in the upstream direction to adjacentupstream nodes in a recursive manner. This performance metric data 502can be sent over a same bidirectional data channel as the batch data insome embodiments. Alternatively, in some embodiments the performancemetric data can be sent over a separate channel from that used topropagate the batch data. As will be discussed in more detail below,reactive nodes use the cumulative performance metrics M_(ij) reported bytheir adjacent downstream reduction nodes—representing downstreamprocessing latencies—to calculate an appropriate data batch size for anext data batch transmission, so that the reactive node can deliver itsdata batch to a downstream reduction node in accordance with thedownstream node's current processing capacities.

In the example depicted in FIG. 5 , S_(ij) denotes a node (a reactive orreduction node), while M_(ij) denotes the performance metric of areduction node S_(ij) (e.g., node S₂₂) that is to be communicated to anadjacent upstream reactive node S_((i-1)k) (e.g., node S₁₁). Performancemetric M_(ij) represents only the local processing latency (given byequation (2) above) of node S_(ij), and does not include cumulativelatencies of nodes that are downstream relative to node S_(ij). That is,local processing latency for a node is given as:M _(ij) =T _(rdr) +T _(rdt) +T _(rdw)  (3)

In a parallel pipeline architecture, a given node system 302 may havemultiple adjacent downstream nodes and/or multiple adjacent upstreamnodes. In the example depicted in FIG. 5 , node S₁₁ is connected tomultiple adjacent downstream nodes S₂₁, S₂₂, and S₂₃, while node S₂₂ isitself connected to multiple adjacent downstream nodes S₃₁, S₃₂, andS₃₃. To ensure that a reactive node S_(ij) sets the size of its outgoingdata batch directed to a reduction node S_(kn) in a manner that takesinto account current downstream processing capacity, the reactive nodeS_(ij) considers both the local processing latency of the targetreduction node S_(kn) itself, as well as the cumulative processinglatencies of nodes that are downstream relative to the reduction nodeS_(kn), since these downstream latencies can impact the total latencyexperienced at the target reduction node S_(kn).

In general, a reactive node S_(ij) can calculate a node cumulativeperformance metric (or cumulative processing latency) CL_(ij,kn) for thepurposes of transmitting a data batch to an adjacent reduction nodeS_(kn) as a sum of the local performance metric (e.g., processinglatency) M_(kn) of the target reduction node S_(kn), the maximum of theperformance metrics (M_(k+1,1), M_(k+1,2), M_(k+1,3), . . . ) of anydownstream nodes (S_(k+1,1), S_(k+1,2), S_(k+1,3), . . . ) that areimmediately adjacent to the target reduction node S_(kn), the maximum ofthe performance metrics (M_(k+2,1), M_(k+2,2), M_(k+2,3), . . . ) of anyfurther downstream nodes (S_(k+2,1), S_(k+2,2), S_(k+2,3), . . . ) thatare immediately adjacent to these adjacent nodes, and so on through thedata pipeline. This can be written as:CL_(ij,kn) =M _(kn)+Σ(max(M _(k+1,1) ,M _(k+1,2) ,M _(k+1,3), . . .),max(M _(k+2,1) ,M _(k+2,2) ,M _(k+2,3), . . . ), . . . )  (4)

In the example pipeline segment depicted in FIG. 5 , if nodes S₃₁, S₃₂,and S₃₃ are assumed to be the final nodes in the data pipeline (e.g.,nodes that deliver data batches to cloud storage or other enddestinations for the data batches), the cumulative performance metricsCL_(11,22) calculated by the reactive node S₁₁ for transmission of adata batch to reduction node S₂₂ is calculated, per equation (4), as:CL_(11,22) =M ₂₂+max(M ₃₁ ,M ₃₂ ,M ₃₃)  (5)

Since calculation of this cumulative performance metric CL_(11,22)requires knowledge of the localized performance metrics M of multipledownstream node systems 302 in the pipeline, the node systems 302 can beconfigured to propagate their performance metric data M to the reactivenode in a recursive manner. For example, in the architecture depicted inFIG. 5 , nodes S₃₁, S₃₂, and S₃₃ can provide their respectiveperformance metrics M₃₁, M₃₂, and M₃₃ to the reduction node S₂₂, whichprovides these downstream performance metrics to the reactive node Stogether with its own localized performance metric M₂₂ so that thereactive node S can determine the cumulative performance metrics (orcumulative processing latency) CL_(11,22) based on these performancemetric values using equation (4) (or a variation thereof). To this end,as part of each data transmission cycle for the pipeline as a whole, theperformance monitoring component 308 of each node system 302 can measurethe local processing latency of its corresponding node system 302 andsend this performance metric information M to adjacent upstream nodesystems 302. Upon receipt of this downstream performance metricinformation, these adjacent upstream node systems 302 can calculatetheir respective cumulative performance metrics CL_(ij,kn) based on thereceived downstream performance metric information in preparation tosend their next data batches 402, and can also relay this downstreamperformance metric information together with their own local performancemetric information to their respective adjacent upstream node systems302.

In some embodiments, the node systems 302 that make up a data pipelinecan be configured to send data batches 402 to their adjacent downstreamnode systems 302 in discrete transmission cycles. In such embodiments,for each data transmission cycle, the node systems 302 can perform anorchestrated handshaking sequence of recursively propagating performancemetric data to upstream node systems, calculating cumulative performancemetrics at each node system 302 that will be acting as a reactive node(that is, each node system 302 that will be sending a data batch 402 toan adjacent downstream node system 302) based on this propagatedperformance metric data, setting the sizes of the next data batches 402that will be sent by each node system 302 for the next data transmissioncycle based each node system's calculated cumulative performance metric,and sending their data batches 402 in accordance with the dynamicallyset batch size. This sequence is described in more detail below.

FIG. 6 is a diagram illustrating the first two steps of an example datatransmission cycle that can be carried out by one or more embodiments ofthe data pipeline node systems 302. FIG. 7 is a diagram illustrating thesubsequent three steps of the transmission cycle. In this example, thesteps are described from the perspective of reactive node S₁₁. It is tobe appreciated, however, that the sequence described herein can becarried out by any of the node systems 302 that will be acting as areactive node during the transmission cycle (that is, any node that willbe sending a data batch 402 to an adjacent downstream node).

In this example, reactive node S₁₁ will be sending a data batch toreduction node S₂₂, which is itself connected to three furtherdownstream nodes S₃₁, S₃₂, and S₃₃. It is assumed that the reactive nodeS₁₁ has received and stored industrial data (e.g., timeseries data) froman adjacent upstream node of the pipeline (not shown in FIG. 6 ) or froman edge device that has collected the data from data tags of one or moreplant-floor industrial devices and fed this collected data to thepipeline. As a general batch size optimization strategy, the reactivenode S₁₁ will calculate a cumulative downstream processing latency basedon the processing metric information received from the reduction node,as discussed above, and set the size of the next outgoing data batchbased on this cumulative processing latency in view of a specifieddesired or expected latency DL. In some scenarios, this desired latencyDL can be predefined by a user (e.g., an administrator of the datapipeline) as an expected latency that will be experienced under normaloperating circumstances, which may be a function of such factors as avolume of data being collected from the industrial devices, the rate atwhich the data is collected, the lengths of the data pipelines, overheadassociated with the cloud-based application or service that will beconsuming the data (e.g., a visualization application, an analyticsapplication, a reporting application, a control application, etc.), orother such factors. Alternatively, a machine learning system can be usedin some configurations to monitor performance of the data pipelines overtime and dynamically infer a latency DL can be expected under normaloperation conditions.

As an initial step of the transmission cycle, the reactive node S₁₁reads or receives reactive node performance metric data 602 from itsadjacent downstream reduction node S₂₂ (step 1 in FIG. 6 ). As discussedabove, this performance metric data 602 comprises the performance metricM₂₂ for the reduction node S₂₂ itself, as well as performance metricsM₃₁, M₃₂, and M₃₃ received from downstream nodes S₃₁, S₃₂, and S₃₃,which are conveyed to the reactive node Sit by the reduction node S₂₂.The various performance metric values are generated by the performancemonitoring components 308 of their respective node systems 302.

Upon receipt of the performance metric data 602, the batch sizecalculation component 310 of reactive node S₁₁ calculates a cumulativeprocessing latency CL based on the received performance metric data 602(step 2 in FIG. 6 ). FIG. 8 is a diagram illustrating calculation of thecumulative processing latency CL. A cumulative latency calculation block802 of the batch size calculation component 310 can calculate thecumulative processing latency CL based on the performance metric data602 received from the reduction node S₂₂ according to equation (4)above, or a variation thereof.

The resulting cumulative processing latency CL is provided to a batchsize calculation block 804 of the batch size calculation component 310,together with an optimization coefficient K. In general, theoptimization coefficient K represents a predefined relationship betweenlatency and batch size. In the present example, this relationship isassumed to be substantially linear. FIG. 9 is a graph 902 representingan example relationship between downstream performance latency relativeto a reactive node and a data batch size that is to be set for thereactive node S₁₁ as a function of the downstream latency. As depictedin graph 902, BS_(ml) is a batch size associated with a maximumperformance latency (BS_(ml) is also considered the minimum batch sizethat can be set by the reactive node), BS₀ is an initial or defaultbatch size associated with the desired or expected performance latencyDL (that is, the batch size BS₀ that will typically be set under normaloperating conditions when the actual cumulative latency CL is within adefined tolerance of the desired or expected latency DL), and ML is themaximum performance latency. If CL represents the actual cumulativeperformance latency reported by the reduction node S₂₂, then BS is thebatch size that is to be set for the next data transmission from thereactive node S₁₁ to the reduction node S₂₂.

As depicted by graph 902, if performance latency and batch size areplotted on the x-axis and y-axis, respectively, the relationship betweenthe two is defined to be linear with a negative slope. This relationshipis based on the understanding that, if the downstream cumulativeperformance latency CL increases (e.g., due to downstream databottlenecks, spikes in data volume, etc.), the size of the data batchsent by the reactive node S₁₁ should be reduced in order to alleviatethe downstream data congestion. Since the relationship betweenperformance latency and batch size is given to be linear in thisexample, the coefficient K is defined as the slope of this relationship,which can be written asK=(BS_(ml)−BS₀)/(ML−DL)<0  (6)or the difference between the initial batch size BS₀ and the batch sizeBS_(ml) associated with the maximum latency, divided by the differencebetween the desired or expected latency DL and the maximum latency ML.Since the slope of this relationship is negative, K is less than zero.This optimization coefficient K, as well as the values of BS_(ml), BS₀,ML, and DL that were used to compute the coefficient K, can be stored onthe reactive node system 302 ₁₁ (S₁₁) for access by the batch sizecalculation component 310.

Although graph 902 depicts a linear relationship between batch size andperformance latency, in some embodiments this relationship may bedefined as a more complicated non-linear relationship. In suchembodiments, the formula for coefficient K given by equation (6) can bereplaced with an equation that reflects the non-linear relationship.

The desired or expected latency DL can represent an expected cumulativeperformance latency for the reduction node S₂₂ under normal operatingconditions given the overall data pipeline application. For example, theexpected latency may be a function of the rate at which data iscollected from the data tags of the plant-floor industrial devices, theprocessing capacity of the node S₂₂, a processing overhead associatedwith an application executed by the node S₂₂ and applied to the incomingdata, or other such factors. In some scenarios, the desired latency DLcan be a fixed value that is preset by an administrator to reflect theexpected latency for the node system 302. Alternatively, in someembodiments, the desired or expected latency DL can be continuously orperiodically re-evaluated and dynamically updated based on measurementand analysis of the node latencies over time. For example, a node system302 or a higher-level analysis system can monitor the performancelatency experienced by the node system 302 and learn an averageperformance latency of the node system 302 over time. In someembodiments, this monitoring and analysis can be carried out by theperformance monitoring component 308 on the node system 302 itself.Alternatively, latency monitoring and analysis can be performed by aseparate higher-level analytic system that monitors latencies atrespective nodes across the entire data pipeline. In either case, oncethe average performance latency over time has been learned, the analysissystem can dynamically set the expected or desired latency DL based onthis learned latency. In an example scenario, based on monitoring andanalysis of the latencies at the reactive node S₂₂ over time, it may bedetermined that the reactive node S₂₂ can process data faster thaninitially expected. Based on this assessment, the batch size calculationcomponent 310 can reduce the value of DL accordingly to bring thedesired latency in line with the measured expected latency. If thedesired latency DL is updated in this manner, batch size calculationcomponent 310 may also update the optimization coefficient K if this newdesired latency changes the relationship between latency and batch size.

Once the batch size calculation component 310 for the reactive node S₁₁has calculated the cumulative performance latency CL based on thedownstream performance metric data 602 that was back-propagated by thereduction node S₂₂, the batch size calculation block 804 of the batchsize calculation component 310 determines a suitable data batch size BSfor the next data transmission to the reduction node S₂₂ based on thecumulative performance latency CL (step 3 in FIG. 7 ). FIG. 10 is adiagram illustrating calculation of the batch size BS by the batch sizecalculation block 804. Given the cumulative performance latency CLreported by the reduction node S₂₂ (e.g., as determined based onequation (4) above) and the relationship between latency and batch sizerepresented by coefficient K (as given by equation (6)), the batch sizecalculation block 804 can calculate a batch size BS according toBS=max{K(CL−DL)+BS₀,BS_(ml)}  (7)or a variation thereof. The values of the batch size BS, initial batchsize BS₀ and maximum latency batch size BS_(ml) can represent anysuitable unit of measure for data size, including but not limited tokilobytes, megabytes, gigabytes, or other units.

In this example, the batch size BS is calculated by determining thedifference between the cumulative performance latency CL reported by thereduction node S₂₂ and the desired or expected latency DL, determiningthe product of this difference and coefficient K, and adding the initialbatch size BS₀ associated with the desired or expected latency DL to theresulting product. If this result is greater than the batch size BS_(ml)associated with the maximum latency ML, this result (or an approximationthereof) is taken as the batch size BS for the next data transmissionfrom the reactive node S₁₁ to the reduction node S₂₂. Alternatively, ifthe result is less than the batch size BS_(ml), then the next batch sizeBS is set to be equal to BS_(ml). As shown by equation (7), the batchsize BS is determined by offsetting the batch size relative to theinitial or default batch size BS₀ associated with the expected latencyDL, where the amount of the offset is based on the difference betweenthe cumulative performance latency CL and the expected latency DL.

Equation (7) is based on the assumed linear relationship between batchsize and performance latency depicted by graph 902. Multiplying thecoefficient K—representing the slope of graph 902—with the differencebetween the reported cumulative latency CL and the desired or expectedlatency DL yields a corresponding negative difference between theinitial or default batch size BS₀ and the batch size BS corresponding tothe reported cumulative latency CL. Adding the initial or default batchsize BS₀ to this result yields the batch size BS corresponding to thecurrently reported cumulative latency CL. This batch size BS is used asthe size of the next outgoing data batch sent by the reactive node S₁₁to the reduction node S₂₂ if BS is greater than the minimum batch sizeBS_(ml) If BS is less than the minimum batch size BS_(ml), then BS_(ml)is used as the next outgoing batch size.

As noted above, although equation (7) assumes a linear relationshipbetween latency and batch size as represented by graph 902, someembodiments of the batch size calculation component 310 can replaceequation (7) with a formula for batch size that assumes a non-linearrelationship between latency and batch size. Such embodiments may beuseful for systems in which it is considered preferable to reduce thebatch size at an increasing rate as the cumulative latency increases(e.g., an exponential relationship), or to maintain the same batch sizefor a range of latencies, and only decrease the batch size if thelatency exceeds a defined threshold (e.g., an incremental steprelationship).

Once the batch size BS for the next outgoing data batch has beencalculated, the reactive node's data output component 306 isreconfigured to output its next data batch in accordance with thecalculated batch size BS. FIG. 11 is a diagram illustrating adjustmentof the reactive node's batch size based on results of the batch sizecalculation. At this stage, it is assumed that the reactive node S₁₁stores queued data 1102 that was previously received from an adjacentupstream node (either another node system 302 or an edge device 106) andis waiting to be sent to the reduction node S₂₂. Batch size calculationcomponent 310 generates batch size data 1002 identifying the batch sizeBS that was calculated based on the cumulative performance latency CLreported by the reduction node S₂₂, as described above. The batch sizecalculation component 310 provides this batch size data 1002 to thebatch size setting component 312, which provides an instruction 1108 tothe data output component 306 to update the size of the next data batchto match the batch size BS reported by the batch size calculationcomponent 310.

In some embodiments, prior to sending out the next batch to thereduction node S₂₂, a timestamp alignment component 314 can first referto time reference data 1110 generated by a timestamp moderator agent anddetermine, based on this time reference data 1110, whether the next databatch 702 is to be released during the current transmission cycle ordelayed until a subsequent transmission cycle (step 4 in FIG. 7 ). Thisstep will be described in more detail below in connection with FIGS.12-15 b.

Upon receipt of the instruction 1108, the data output component 306 setsthe size of its next outgoing data batch 702 to match the batch size BSdefined by the instruction 1108 (as calculated by the batch sizecalculation component 310). Subsequently, when it is time for thereactive node S₁₁ to send out its next data batch 702 to the adjacentreactive node S₂₂, the data output component 306 adds an amount of thequeued data 1102 equal to the batch size BS to the data batch 702 andsends the resulting data batch 702 to the reactive node S₂₂ (step 5 inFIG. 7 ). In some embodiments, the data output component 306 can add theoldest subset of the queued data 1102 to the data batch 702 in afirst-in-first-out manner, ensuring that the data that has been waitingin the data queue the longest is given transmission priority over newerdata.

In some embodiments, the sequence described above—comprising steps 1-5depicted in FIGS. 6 and 7 and elaborated in FIGS. 8-11 —can be executedfor each data transmission cycle of the data pipeline. Alternatively, insome embodiments the steps of back-propagating performance metric data602, calculating cumulative performance metrics CL at each reactivenode, and re-computing a suitable batch size BS may be performed lessthan every cycle. In the latter case, the batch size may be recalculatedon a periodic time basis (e.g., every 30 seconds, every minute, etc.),on the basis of every Nth data transmission cycle (e.g., every 5^(th)cycle, every 10^(th) cycle, etc.), or in response to a predefinedcondition (e.g., a determination that latency at one or more nodesystems 302 has exceeded a defined level, detection of an event that maybe indicative of a latency issue, etc.).

As noted above, some embodiments of the pipeline node system 302 caninclude a timestamp alignment component 314 that refers to a timestampreference generated by a timestamp moderator agent prior to sending outthe next data batch 702, such that the timing of delivery of the databatch 702 is determined in part by this timestamp reference (step 4 ofFIG. 7 ). This can ensure that queued data stored on the various nodesystems 302 is delivered to a final destination in a substantiallysynchronized manner even when some of the data is delayed at one or moreof the node systems 302. FIGS. 12 and 13 illustrate this synchronizeddelivery control approach. In the illustrated example, a number ofpipeline node systems 302 are chained together to form multiple paralleldata pipelines that deliver industrial data to a target destination1206. The target destination 1206 may be, for example, a cloud-basedapplication that stores and processes the data received from thepipelines, such as a cloud-based visualization system, an analyticsystem, a reporting system, or other such applications. In the examplearchitecture depicted in FIG. 12 , the chained node systems 302 formfive data pipelines of different lengths—a first pipeline through nodesS₁₁ and S₂₁; a second pipeline through nodes S₁₁, S₂₂, and S₃₁; a thirdpipeline through nodes S₁₁, S₂₂, and S₃₂; a fourth pipeline throughnodes S₁₁, S₂₂, and S₃₃; and a fifth pipeline through nodes S₁₁ and S₂₃.

The node systems 302 in such an architecture can be categorizedaccording to layer, where a node's layer represents the number ofpipeline segments between the node and the edge device (e.g., edgedevice 106 in FIG. 1 , not shown in FIG. 13 ) from which the industrialdata originates. In the illustrated example, node S₁₁ is assumed to bethe node that receives timestamped industrial data from an edge device(not shown in FIG. 12 ), and as such is classified as a Layer 1 node.Nodes S₂₁, S₂₂, and S₂₃ are all adjacent to the Layer 1 node S₁₁ and aretherefore classified as Layer 2 nodes. Any nodes that are adjacent to aLayer 2 node—such as nodes S₃₁, S₃₂, and S₃₃—are classified as Layer 3nodes. These layer designations are propagated through the entirelengths of the data pipelines.

In many data pipeline applications, items of industrial data enteringthe pipeline are timestamped to indicate the time at which the data wasmeasured or generated. These timestamps are used by the destinationapplications to ensure accurate analysis or visualization of thecollected data. For example, visualization applications that render thetime-series data in a specified format (e.g., as a time-series graph, asa time-based animation, etc.) rely on these timestamps to correctlydisplay the temporal relationships between events on the plant floor.Similarly, analytic applications use the timestamps to accuratelyidentify orders of events or causal relationships between events.

When latencies at one or more node systems 302 result in excessivebuffering of data at one or more of the node systems 302, andparticularly when these latencies disproportionately affect differentpipelines in the chain, timestamped data from the various data pipelinesmay be received at the target destination 1206 at different speeds. As aresult, the data may not be received at the target destination 1206 insynchronization with the timestamps. That is, one data pipeline that isnot experiencing excessive latency may deliver industrial data with arecent timestamp at the same time that a slower pipeline is deliveringindustrial data with an older timestamp due to latency-related delays.This lack of data synchronization can cause problems with applicationsat the target destination 1206 that consume this data. For example,visualization of received data values by a visualization application mayfall out of synchronization with the timestamps since the data from thevarious pipelines that feeds this application is arriving at differentspeeds.

To address this issue, a timestamp moderator agent 1202 can monitor thetimestamps of queued data currently stored on the node systems 302 andcoordinate delivery of data batches 702 by the nodes of each layer suchthat data is received at the target destination 1206 in a substantiallysynchronized manner FIG. 14 is a block diagram of an example timestampmoderator agent 1202 according to one or more embodiments of thisdisclosure. The timestamp moderator agent 1202 can constitutemachine-executable components embodied within machine(s), e.g., embodiedin one or more computer-readable mediums (or media) associated with oneor more machines. Such components, when executed by one or moremachines, e.g., computer(s), computing device(s), automation device(s),virtual machine(s), etc., can cause the machine(s) to perform theoperations described.

Timestamp moderator agent 1202 can include a node interface component1404, a timestamp monitoring component 1406, a timestamp referencecomponent 1408, one or more processors 1418, and memory 1420. In variousembodiments, one or more of the node interface component 1404, timestampmonitoring component 1406, timestamp reference component 1408, the oneor more processors 1418, and memory 1420 can be electrically and/orcommunicatively coupled to one another to perform one or more of thefunctions of the timestamp moderator agent 1202. In some embodiments,components 1404, 1402, and 1408 can comprise software instructionsstored on memory 1420 and executed by processor(s) 1418. Timestampmoderator agent 1202 may also interact with other hardware and/orsoftware components not depicted in FIG. 14 . For example, processor(s)1418 may interact with one or more external user interface devices, suchas a keyboard, a mouse, a display monitor, a touchscreen, or other suchinterface devices.

Node interface component 1404 can be configured to exchange data withpipeline node systems 302 that make up one or more IIoT data pipelines.For example, node interface component 1404 can receive timestampinformation indicating the timestamps of oldest data still queued oneach node system 302, and deliver layer-specific time reference data tothe node systems 302 which informs the nodes whether they should releasetheir next data batch 702 or if delivery of the data batch 702 should bedelayed. Timestamp monitoring component 1406 can be configured tomonitor the timestamp data received from the node systems 302 togetherwith the node layer from which each item of timestamp data was received.Timestamp reference component 1408 can be configured to generate, foreach layer of the pipeline, a time reference that can be used toidentify a timestamp or timestamp range that is permitted to be releasedin a current data transmission cycle. This time reference is generatedbased on the timestamp data monitored by the timestamp monitoringcomponent 1406.

The one or more processors 1418 can perform one or more of the functionsdescribed herein with reference to the systems and/or methods disclosed.Memory 1420 can be a computer-readable storage medium storingcomputer-executable instructions and/or information for performing thefunctions described herein with reference to the systems and/or methodsdisclosed.

FIG. 12 illustrates monitoring of timestamp data 1204 by the timestampmoderator agent 1202. The timestamp data 1204 can be generated by eachnode's timestamp alignment component 314 and collected by the agent'snode interface component 1404. Timestamp data 1204 reported by a nodesystem 302 identifies a timestamp or range of timestamps associated withthe oldest data currently queued on the node and awaiting transmissionto the next downstream node (or target destination 1206). In someembodiments, timestamp moderator agent 1202 can collect this timestampdata 1204 each data transmission cycle so that the layer-specific timereferences can be updated each cycle.

The timestamp monitoring component 1406 of the timestamp moderator agent1202 segregates this timestamp data 1204 according to the node layerfrom which the timestamp data 1204 was received. For each layer, thetimestamp reference component 1408 of the timestamp moderator agent 1202compares the timestamp data 1204 for the node systems 302 in that layerto identify the oldest queued data reported for the layer. If thetimestamp for the oldest queued data in a node system 302 lagssignificantly behind the timestamps of the data queued in other nodesystems 302 within the same layer—e.g., due to excessive latency causingthe data in the node's pipeline to be propagated at a slower speedrelative to the other nodes in the layer—the data traveling through thelayer is likely to arrive at the target destination 1206 in anon-synchronous manner if not corrected. That is, data with oldertimestamps will arrive at the target destination at the same time asdata with more recent timestamps.

To address this issue, timestamp moderator agent 1202 can resynchronizethe data by permitting the node system 302 with the oldest queued datato release its data batch 702 in the current data transmission cyclewhile instructing the other nodes in the same layer to delaytransmission of their respective data batches 702 until a subsequenttransmission cycle. This results in a realignment or substantialrealignment of the timestamps of the queued data across all nodes of thelayer.

In order to regulate transmission of data batches 702 to maintainalignment of timestamps within each layer, the timestamp moderator agent1202 can generate and deliver layer-specific time references to eachnode system 302 based on analysis of the timestamp data 1204. FIG. 13 isa diagram illustrating delivery of layer-specific time references to thenode systems 302 that make up the data pipelines. As noted above, thetimestamp moderator agent 1202 examines the timestamps associated withthe oldest queued data (e.g., queued data 1102 in FIG. 11 ) stored oneach node system 302, as reported by timestamp data 1204. For each layerof the pipeline network, the timestamp moderator agent 1202 identifies,based on this examination, the timestamp corresponding to the oldestqueued data among all the node systems 302 within that layer. Unless thetimestamps across the nodes 302 of the layer are already substantiallysynchronized, the oldest timestamp queued within the layer will bereported by a node whose oldest timestamps lag behind the timestamps ofthe oldest data in the other nodes of the same layer.

Based on results of this determination, the timestamp referencecomponent 1404 generates time reference data 1110 for each layer thatmakes up the pipeline network. Time reference data 1110 specifies areference timestamp {ref_(ij)} that can be used by each node system 302in the layer to determine whether the node system 302 will send its databatch 702 in the current data transmission cycle, or will delaytransmission of its data batch 702 until a subsequent data transmissioncycle. In an example embodiment, the reference timestamp {ref_(ij)} fora given layer can indicate the oldest timestamp among the timestamp data1204 reported by the nodes in that layer. The node interface component1404 of the timestamp moderator agent 1202 can send this time referencedata 1110 to each node system 302 in the layer.

Returning briefly to FIG. 11 , the time reference data 1110 is examinedby the timestamp alignment component 314 of each node system 302. Ingeneral, the timestamp alignment component 314 will issue a releasepermissive 1106 to the data output component 306 of the node system 302if the oldest queued data 1102 stored on the node system 302 has atimestamp that matches, or is older than, the reference timestamp{ref_(ij)} indicated by time reference data 1110. For the currentpipeline transmission cycle, the data output component 306 will onlysend the data batch 702 during the current cycle if the timestampalignment component 314 issues the release permissive 1106.Alternatively, if the queued data 1102 only comprises data havingtimestamps that are more recent than the reference timestamp {ref_(ij)}indicated by the time reference data 1110, in excess of a defined lagtolerance, the timestamp alignment component 314 will not issue therelease permissive 1106 and the data output component 306 will not sendthe data batch 702 in the current transmission cycle.

FIGS. 15 a and 15 b illustrate how this approach can bring thetimestamped data within a given layer into alignment after the data inone of the node systems 302 begins to lag behind the data in the othernode systems 302 in the layer (e.g., due to abnormal latenciesexperienced by the lagging node). In this example, the nodes in Layer 2(nodes S₂₁, S₂₂, and S₂₃) store queued data 1102 ₂₁, 1102 ₂₂, and 1102₂₃, respectively. Nodes S₂₁ and S₂₂ have experienced no excessivelatency within their associated data pipelines, and as such thetimestamps of the oldest data in their data queues 1102 ₂₁ and 1102 ₂₂are substantially equal and given as time T_(2j)=t. By contrast, thequeued data 1102 ₂₃ stored on node S₂₃ has been released at a slowerrate relative to nodes S₂₁ and S₂₂ due to unexpected downstreamlatencies, and as a result the timestamp of the oldest data in thequeued data 1102 ₂₃ lags behind the timestamps of the data in the otherqueued data 1102 ₂₁ and 1102 ₂₂. This lagged timestamp is given byT₂₃=(t−5) in this example.

As discussed above in connection with FIG. 12 , the node systems 302report their oldest queued timestamps T_(2j) as timestamp data 1204 tothe timestamp moderator agent 1202. The timestamp moderator agent 1202determines that the timestamp T₂₃ reported by node S₂₃ is the oldest ofthe timestamps reported by the nodes of that layer, and therefore issuestime reference data 1110 _(2j) to the nodes of the layer indicating that{ref_(2j)}=(t−5) is the current time reference for the layer.

The timestamp alignment component 314 of each node in the layer receivesthis time reference data 1110 _(2j) and determines whether to issue therelease permissive 1106 to its corresponding data output component 306based on a comparison between the time reference {ref_(2j)}=(t−5) andthe timestamp of the oldest data in the node's queued data 1102. Asnoted above in connection with FIG. 11 , the timestamp alignmentcomponent 314 will issue the release permissive 1106 if the oldesttimestamp in the node's queued data 1102 is equal to or older than thetime reference {ref_(2j)}=(t−5). The timestamp alignment component 314may also issue the release permissive 1106 if the oldest timestamp inthe node's queued data 1102 is more recent than the {ref_(2j)}=(t−5),but the difference between the oldest timestamp and the time reference{ref_(2j)} does not exceed a defined lag time. This ensures that databatches 702 are only delayed if the timestamp lag between two or more ofthe node systems 302 becomes sufficiently large to merit delaying somedata queues in order to realign the timestamps.

FIG. 15 b is a diagram illustrating selective transmission of a databatch 702 in order to realign the timestamps in the example depicted inFIG. 15 a . Nodes S₂₁ and S₂₂ compare the time reference{ref_(2j)}=(t−5) with the timestamps of the oldest data in theirrespective queued data 1102 ₂₁ and 1102 ₂₂. Based on a determinationthat the oldest timestamps in their queued data 1102 ₂₁ and 1102 ₂₂exceed the time reference {ref_(2j)}=(t−5) in excess of the defined lagtime, the timestamp alignment component 314 of nodes S₂₁ and S₂₂ refrainfrom issuing a release permissive 1106, and consequently nodes S₂₁ andS₂₂ do not send their data batches 702 to their adjacent reactive nodes.In contrast, node S₂₃ determines that the oldest timestamp in its queueddata 1102 ₂₃ is equal to or less than the time reference{ref_(2j)}=(t−5) and releases its data batch 702 accordingly. Releasingthis data batch 702 causes the data that was included in the data batch702 to be deleted from the queued data 1102 ₂₃, and consequently theoldest timestamp in the remaining queued data 1102 ₂₃ are shiftedforward in time. The amount of this shift depends on the batch size BSof the data batch 702 and the time density of the queued data 1102 ₂₃.In the illustrated example, the resulting timestamp shift causes theoldest timestamp on node S₂₃ to shift from T₂₃=(t−5) to T₂₃=t. Sincethis timestamp shift occurs only on node S₂₃ but not on nodes S₂₁ andS₂₂— which have delayed transmission of their respective data batches702 and therefore did not experience a timestamp shift in the presenttransmission—the timestamps across the three nodes in the layer arebrought into alignment. Accordingly, when the time reference {ref_(2j)}is updated during the next transmission cycle, all three nodes in thelayer will be permitted to transmit their data batches 702, since theoldest timestamps in the three sets of queued data 1102 ₂₁, 1102 ₂₂, and1102 ₂₃ will satisfy the time reference {ref_(2j)}.

The use of a timestamp moderator agent 1202 as described above inconnection with FIGS. 11-15 b can ensure that lagged data within a layerof a parallel pipeline network is delivered before other data in thelayer, thereby synchronizing delivery of data to the target destination1206 via the parallel pipelines even if one or more of the pipelinesexperiences excessive unexpected latency.

The dynamic IIoT reactive buffering and batch size regulation featuresdescribed herein offering a mechanism for seamlessly auto regulating thetransmission of data between nodes of a data pipeline. This pipelinedata flow management does not require application-level control, butrather regulates control based purely on performance and data priority.By establishing an inter-node handshaking sequence in which downstreamlatency is reported to each node and dynamically adjusting the sizes ofoutgoing data batches in response to changes in cumulative downstreamlatency, the system described herein can automatically alleviate datacongestion by reducing the size of a node's outgoing data batch whendownstream performance latencies increase, thereby preventing thedownstream nodes from becoming overwhelmed with data during busyperiods.

Node systems 302 having dynamic batch size adjustment features can beused within an architecture that is also regulated by a timestampmoderator agent 1202, as depicted in FIGS. 12 and 13 . This combinedarchitecture can both alleviate data congestion and maintainsynchronized delivery of data to an end destination even in the event ofasymmetric data latencies across different parallel pipelines. However,the timestamp moderator agent 1202 and node systems 302 can also be usedindependently in some architectures without departing from the scope ofone or more embodiments. That is, the timestamp moderator agent 1202 canbe used in parallel pipeline architectures that do not support dynamicadjustment of data batch sizes, while the node systems 302 can be usedwithin a data pipeline architecture that is not regulated by a timestampmoderator agent 1202.

FIGS. 16-19 illustrates methodologies in accordance with one or moreembodiments of the subject application. While, for purposes ofsimplicity of explanation, the methodologies shown herein is shown anddescribed as a series of acts, it is to be understood and appreciatedthat the subject innovation is not limited by the order of acts, as someacts may, in accordance therewith, occur in a different order and/orconcurrently with other acts from that shown and described herein. Forexample, those skilled in the art will understand and appreciate that amethodology could alternatively be represented as a series ofinterrelated states or events, such as in a state diagram. Moreover, notall illustrated acts may be required to implement a methodology inaccordance with the innovation. Furthermore, interaction diagram(s) mayrepresent methodologies, or methods, in accordance with the subjectdisclosure when disparate entities enact disparate portions of themethodologies. Further yet, two or more of the disclosed example methodscan be implemented in combination with each other, to accomplish one ormore features or advantages described herein.

FIG. 16 illustrates an example methodology 1600 for regulating a size ofa data batch sent by a reactive node of an industrial IIoT data pipelineto an adjacent downstream reactive node. The methodology 1600 can beperformed for each data transmission cycle of a reactive node of thepipeline. Initially, at 1602, data is received and queued at a reactivenode of an IIoT data pipeline. The data can be received from an adjacentupstream node of the pipeline, or from an edge device that collectsindustrial data from industrial devices and feeds the data to thepipeline. At 1604, performance metric data is received at the reactivenode from an adjacent downstream reduction node of the data pipeline.The performance metric data can convey a current processing latency ofthe reduction node itself, as measured by the reduction node, as well asinformation regarding processing latencies of nodes that are downstreamrelative to the reduction node.

At 1606, a cumulative downstream latency is calculated at the reactivenode based on the performance metric data received at step 1604. At1608, a batch size for a next data batch to be sent from the reactivenode to the reduction node is calculated based on the cumulativedownstream latency calculated at step 1606. At 1610, the size of thenext data batch to be sent by the reactive node is set in accordancewith the batch size calculated at step 1608. At 1612, an amount of thequeued data equal to or less than the calculated batch size is sent tothe reduction node.

FIG. 17 illustrates an example methodology 1700 for both regulating asize of a data batch sent by a reactive node of a parallel data pipelinenetwork to an adjacent downstream reactive node, as well as maintainingsynchronization of timestamped data within respective node layers of theparallel data pipeline network. Initially, at 1702, data is received andqueued at a reactive node from an adjacent upstream node (similar tostep 1602 of methodology 1600). At 1704, performance metric data isreceived at the reactive node from an adjacent downstream reduction node(similar to step 1704 of methodology 1600). At 1706, a cumulativedownstream latency is calculated based on this performance metric data(similar to step 1606 of methodology 1600). At 1708, a batch size forthe next data batch to be sent by the reactive node is calculated basedon the cumulative downstream latency (similar to step 1608 ofmethodology 1600). At 1710, the size of the next data batch is set inaccordance with the batch size calculated at step 1708.

At 1712, a determination is made as to whether a timestamp associatedwith the oldest data currently queued on the reactive node satisfies acriterion relative to a time reference generated by a time moderatoragent. The time reference can specify a timestamp corresponding to theoldest queued data within a node layer in which the reactive noderesides. In an example configuration, the timestamp of the oldest datasatisfies the criterion if the timestamp is not later than the timestampindicated by the time reference in excess of a defined lag time. If thetimestamp of the oldest data in the data queue satisfies the criterion(YES at step 1712), the methodology proceeds to step 1714, where anamount of the queued data equal to or less than the calculated batchsize is sent form the reaction node to the reduction node (similar tostep 1612 of methodology 1600). Alternatively, if the timestamp of theoldest data in the data queue does not satisfy the criterion (NO at step1712), the data batch is not sent and the methodology returns to step1702 for the next transmission cycle.

FIG. 18 illustrates an example methodology 1800 for maintainingsynchronization of data stamps within a node layer of network ofparallel IIoT data pipelines. Initially, at 1802, data is received andqueued at a reactive node that is part of the network of parallel IIoTdata pipelines. At 1804, a reference timestamp for a current datatransmission cycle is received from a timestamp moderator agent thatmonitors timestamps of data currently queued at each node of thepipeline network. The timestamp reference is specific to the node layerof the parallel pipeline network in which the node resides and regulatestransmission of data batches from the nodes in that layer.

At 1806, a determination is made as to whether the oldest data in thedata queue has a timestamp that satisfies a criterion relative to thereference timestamp received at step 1804. The timestamp of the oldestdata can be considered as having satisfied the criterion if thetimestamp of the oldest data is not later than the timestamp indicatedby the reference timestamp in excess of a defined lag time. If theoldest data has a timestamp that satisfies the criterion (YES at step1806), the methodology proceeds to step 1808, where at least a subset ofthe queued data is sent to an adjacent downstream reduction node of thepipeline network. Alternatively, if the oldest data has a timestamp thatdoes not satisfy the criterion (NO at step 1806), the methodologyproceeds to step 1810, where sending of any of the queued data to theadjacent downstream reduction node is delayed until a subsequenttransmission cycle.

FIG. 19 illustrates an example methodology 1900 for generating areference timestamp to be used by nodes of a layer of a parallel IIoTdata pipeline network to maintain synchronization of timestamps withinthe nodes. Initially, at 1902, timestamp data is received from nodesthat make up a network of parallel IIoT data pipelines. The timestampdata identifies timestamps of the oldest data queued on the respectivenodes. At 1904, for each node layer of the network of data pipelines, areference timestamp is determined that corresponds to the earliesttimestamp identified by the timestamp data received at step 1902. At1906, the reference timestamp determined at step 1904 for each nodelayer is sent to the subset of the nodes within that layer.

Embodiments, systems, and components described herein, as well ascontrol systems and automation environments in which various aspects setforth in the subject specification can be carried out, can includecomputer or network components such as servers, clients, programmablelogic controllers (PLCs), automation controllers, communicationsmodules, mobile computers, on-board computers for mobile vehicles,wireless components, control components and so forth which are capableof interacting across a network. Computers and servers include one ormore processors—electronic integrated circuits that perform logicoperations employing electric signals—configured to execute instructionsstored in media such as random access memory (RAM), read only memory(ROM), a hard drives, as well as removable memory devices, which caninclude memory sticks, memory cards, flash drives, external hard drives,and so on.

Similarly, the term PLC or automation controller as used herein caninclude functionality that can be shared across multiple components,systems, and/or networks. As an example, one or more PLCs or automationcontrollers can communicate and cooperate with various network devicesacross the network. This can include substantially any type of control,communications module, computer, Input/Output (I/O) device, sensor,actuator, and human machine interface (HMI) that communicate via thenetwork, which includes control, automation, and/or public networks. ThePLC or automation controller can also communicate to and control variousother devices such as standard or safety-rated I/O modules includinganalog, digital, programmed/intelligent I/O modules, other programmablecontrollers, communications modules, sensors, actuators, output devices,and the like.

The network can include public networks such as the internet, intranets,and automation networks such as control and information protocol (CIP)networks including DeviceNet, ControlNet, safety networks, andEthernet/IP. Other networks include Ethernet, DH/DH+, Remote I/O,Fieldbus, Modbus, Profibus, CAN, wireless networks, serial protocols,and so forth. In addition, the network devices can include variouspossibilities (hardware and/or software components). These includecomponents such as switches with virtual local area network (VLAN)capability, LANs, WANs, proxies, gateways, routers, firewalls, virtualprivate network (VPN) devices, servers, clients, computers,configuration tools, monitoring tools, and/or other devices.

In order to provide a context for the various aspects of the disclosedsubject matter, FIGS. 20 and 21 as well as the following discussion areintended to provide a brief, general description of a suitableenvironment in which the various aspects of the disclosed subject mattermay be implemented. While the embodiments have been described above inthe general context of computer-executable instructions that can run onone or more computers, those skilled in the art will recognize that theembodiments can be also implemented in combination with other programmodules and/or as a combination of hardware and software.

Generally, program modules include routines, programs, components, datastructures, etc., that perform particular tasks or implement particularabstract data types. Moreover, those skilled in the art will appreciatethat the inventive methods can be practiced with other computer systemconfigurations, including single-processor or multiprocessor computersystems, minicomputers, mainframe computers, Internet of Things (IoT)devices, distributed computing systems, as well as personal computers,hand-held computing devices, microprocessor-based or programmableconsumer electronics, and the like, each of which can be operativelycoupled to one or more associated devices.

The illustrated embodiments herein can be also practiced in distributedcomputing environments where certain tasks are performed by remoteprocessing devices that are linked through a communications network. Ina distributed computing environment, program modules can be located inboth local and remote memory storage devices.

Computing devices typically include a variety of media, which caninclude computer-readable storage media, machine-readable storage media,and/or communications media, which two terms are used herein differentlyfrom one another as follows. Computer-readable storage media ormachine-readable storage media can be any available storage media thatcan be accessed by the computer and includes both volatile andnonvolatile media, removable and non-removable media. By way of example,and not limitation, computer-readable storage media or machine-readablestorage media can be implemented in connection with any method ortechnology for storage of information such as computer-readable ormachine-readable instructions, program modules, structured data orunstructured data.

Computer-readable storage media can include, but are not limited to,random access memory (RAM), read only memory (ROM), electricallyerasable programmable read only memory (EEPROM), flash memory or othermemory technology, compact disk read only memory (CD-ROM), digitalversatile disk (DVD), Blu-ray disc (BD) or other optical disk storage,magnetic cassettes, magnetic tape, magnetic disk storage or othermagnetic storage devices, solid state drives or other solid statestorage devices, or other tangible and/or non-transitory media which canbe used to store desired information. In this regard, the terms“tangible” or “non-transitory” herein as applied to storage, memory orcomputer-readable media, are to be understood to exclude onlypropagating transitory signals per se as modifiers and do not relinquishrights to all standard storage, memory or computer-readable media thatare not only propagating transitory signals per se.

Computer-readable storage media can be accessed by one or more local orremote computing devices, e.g., via access requests, queries or otherdata retrieval protocols, for a variety of operations with respect tothe information stored by the medium.

Communications media typically embody computer-readable instructions,data structures, program modules or other structured or unstructureddata in a data signal such as a modulated data signal, e.g., a carrierwave or other transport mechanism, and includes any information deliveryor transport media. The term “modulated data signal” or signals refersto a signal that has one or more of its characteristics set or changedin such a manner as to encode information in one or more signals. By wayof example, and not limitation, communication media include wired media,such as a wired network or direct-wired connection, and wireless mediasuch as acoustic, RF, infrared and other wireless media.

With reference again to FIG. 20 the example environment 2000 forimplementing various embodiments of the aspects described hereinincludes a computer 2002, the computer 2002 including a processing unit2004, a system memory 2006 and a system bus 2008. The system bus 2008couples system components including, but not limited to, the systemmemory 2006 to the processing unit 2004. The processing unit 2004 can beany of various commercially available processors. Dual microprocessorsand other multi-processor architectures can also be employed as theprocessing unit 2004.

The system bus 2008 can be any of several types of bus structure thatcan further interconnect to a memory bus (with or without a memorycontroller), a peripheral bus, and a local bus using any of a variety ofcommercially available bus architectures. The system memory 2006includes ROM 2010 and RAM 2012. A basic input/output system (BIOS) canbe stored in a non-volatile memory such as ROM, erasable programmableread only memory (EPROM), EEPROM, which BIOS contains the basic routinesthat help to transfer information between elements within the computer2002, such as during startup. The RAM 2012 can also include a high-speedRAM such as static RAM for caching data.

The computer 2002 further includes an internal hard disk drive (HDD)2014 (e.g., EIDE, SATA), one or more external storage devices 2016(e.g., a magnetic floppy disk drive (FDD) 2016, a memory stick or flashdrive reader, a memory card reader, etc.) and an optical disk drive 2020(e.g., which can read or write from a CD-ROM disc, a DVD, a BD, etc.).While the internal HDD 2014 is illustrated as located within thecomputer 2002, the internal HDD 2014 can also be configured for externaluse in a suitable chassis (not shown). Additionally, while not shown inenvironment 2000, a solid state drive (SSD) could be used in additionto, or in place of, an HDD 2014. The HDD 2014, external storagedevice(s) 2016 and optical disk drive 2020 can be connected to thesystem bus 2008 by an HDD interface 2024, an external storage interface2026 and an optical drive interface 2028, respectively. The interface2024 for external drive implementations can include at least one or bothof Universal Serial Bus (USB) and Institute of Electrical andElectronics Engineers (IEEE) 1394 interface technologies. Other externaldrive connection technologies are within contemplation of theembodiments described herein.

The drives and their associated computer-readable storage media providenonvolatile storage of data, data structures, computer-executableinstructions, and so forth. For the computer 2002, the drives andstorage media accommodate the storage of any data in a suitable digitalformat. Although the description of computer-readable storage mediaabove refers to respective types of storage devices, it should beappreciated by those skilled in the art that other types of storagemedia which are readable by a computer, whether presently existing ordeveloped in the future, could also be used in the example operatingenvironment, and further, that any such storage media can containcomputer-executable instructions for performing the methods describedherein.

A number of program modules can be stored in the drives and RAM 2012,including an operating system 2030, one or more application programs2032, other program modules 2034 and program data 2036. All or portionsof the operating system, applications, modules, and/or data can also becached in the RAM 2012. The systems and methods described herein can beimplemented utilizing various commercially available operating systemsor combinations of operating systems.

Computer 2002 can optionally comprise emulation technologies. Forexample, a hypervisor (not shown) or other intermediary can emulate ahardware environment for operating system 2030, and the emulatedhardware can optionally be different from the hardware illustrated inFIG. 20 . In such an embodiment, operating system 2030 can comprise onevirtual machine (VM) of multiple VMs hosted at computer 2002.Furthermore, operating system 2030 can provide runtime environments,such as the Java runtime environment or the .NET framework, forapplication programs 2032. Runtime environments are consistent executionenvironments that allow application programs 2032 to run on anyoperating system that includes the runtime environment. Similarly,operating system 2030 can support containers, and application programs2032 can be in the form of containers, which are lightweight,standalone, executable packages of software that include, e.g., code,runtime, system tools, system libraries and settings for an application.

Further, computer 2002 can be enable with a security module, such as atrusted processing module (TPM). For instance with a TPM, bootcomponents hash next in time boot components, and wait for a match ofresults to secured values, before loading a next boot component. Thisprocess can take place at any layer in the code execution stack ofcomputer 2002, e.g., applied at the application execution level or atthe operating system (OS) kernel level, thereby enabling security at anylevel of code execution.

A user can enter commands and information into the computer 2002 throughone or more wired/wireless input devices, e.g., a keyboard 2038, a touchscreen 2040, and a pointing device, such as a mouse 2042. Other inputdevices (not shown) can include a microphone, an infrared (IR) remotecontrol, a radio frequency (RF) remote control, or other remote control,a joystick, a virtual reality controller and/or virtual reality headset,a game pad, a stylus pen, an image input device, e.g., camera(s), agesture sensor input device, a vision movement sensor input device, anemotion or facial detection device, a biometric input device, e.g.,fingerprint or iris scanner, or the like. These and other input devicesare often connected to the processing unit 2004 through an input deviceinterface 2044 that can be coupled to the system bus 2008, but can beconnected by other interfaces, such as a parallel port, an IEEE 1394serial port, a game port, a USB port, an IR interface, a BLUETOOTH®interface, etc.

A monitor 2044 or other type of display device can be also connected tothe system bus 2008 via an interface, such as a video adapter 2046. Inaddition to the monitor 2044, a computer typically includes otherperipheral output devices (not shown), such as speakers, printers, etc.

The computer 2002 can operate in a networked environment using logicalconnections via wired and/or wireless communications to one or moreremote computers, such as a remote computer(s) 2048. The remotecomputer(s) 2048 can be a workstation, a server computer, a router, apersonal computer, portable computer, microprocessor-based entertainmentappliance, a peer device or other common network node, and typicallyincludes many or all of the elements described relative to the computer2002, although, for purposes of brevity, only a memory/storage device2050 is illustrated. The logical connections depicted includewired/wireless connectivity to a local area network (LAN) 2052 and/orlarger networks, e.g., a wide area network (WAN) 2054. Such LAN and WANnetworking environments are commonplace in offices and companies, andfacilitate enterprise-wide computer networks, such as intranets, all ofwhich can connect to a global communications network, e.g., theInternet.

When used in a LAN networking environment, the computer 2002 can beconnected to the local network 2052 through a wired and/or wirelesscommunication network interface or adapter 2056. The adapter 2056 canfacilitate wired or wireless communication to the LAN 2052, which canalso include a wireless access point (AP) disposed thereon forcommunicating with the adapter 2056 in a wireless mode.

When used in a WAN networking environment, the computer 2002 can includea modem 2058 or can be connected to a communications server on the WAN2054 via other means for establishing communications over the WAN 2054,such as by way of the Internet. The modem 2058, which can be internal orexternal and a wired or wireless device, can be connected to the systembus 2008 via the input device interface 2042. In a networkedenvironment, program modules depicted relative to the computer 2002 orportions thereof, can be stored in the remote memory/storage device2050. It will be appreciated that the network connections shown areexample and other means of establishing a communications link betweenthe computers can be used.

When used in either a LAN or WAN networking environment, the computer2002 can access cloud storage systems or other network-based storagesystems in addition to, or in place of, external storage devices 2016 asdescribed above. Generally, a connection between the computer 2002 and acloud storage system can be established over a LAN 2052 or WAN 2054e.g., by the adapter 2056 or modem 2058, respectively. Upon connectingthe computer 2002 to an associated cloud storage system, the externalstorage interface 2026 can, with the aid of the adapter 2056 and/ormodem 2058, manage storage provided by the cloud storage system as itwould other types of external storage. For instance, the externalstorage interface 2026 can be configured to provide access to cloudstorage sources as if those sources were physically connected to thecomputer 2002.

The computer 2002 can be operable to communicate with any wirelessdevices or entities operatively disposed in wireless communication,e.g., a printer, scanner, desktop and/or portable computer, portabledata assistant, communications satellite, any piece of equipment orlocation associated with a wirelessly detectable tag (e.g., a kiosk,news stand, store shelf, etc.), and telephone. This can include WirelessFidelity (Wi-Fi) and BLUETOOTH® wireless technologies. Thus, thecommunication can be a predefined structure as with a conventionalnetwork or simply an ad hoc communication between at least two devices.

FIG. 21 is a schematic block diagram of a sample computing environment2100 with which the disclosed subject matter can interact. The samplecomputing environment 2100 includes one or more client(s) 2102. Theclient(s) 2102 can be hardware and/or software (e.g., threads,processes, computing devices). The sample computing environment 2100also includes one or more server(s) 2104. The server(s) 2104 can also behardware and/or software (e.g., threads, processes, computing devices).The servers 2104 can house threads to perform transformations byemploying one or more embodiments as described herein, for example. Onepossible communication between a client 2102 and servers 2104 can be inthe form of a data packet adapted to be transmitted between two or morecomputer processes. The sample computing environment 2100 includes acommunication framework 2106 that can be employed to facilitatecommunications between the client(s) 2102 and the server(s) 2104. Theclient(s) 2102 are operably connected to one or more client datastore(s) 2108 that can be employed to store information local to theclient(s) 2102. Similarly, the server(s) 2104 are operably connected toone or more server data store(s) 2110 that can be employed to storeinformation local to the servers 2104.

What has been described above includes examples of the subjectinnovation. It is, of course, not possible to describe every conceivablecombination of components or methodologies for purposes of describingthe disclosed subject matter, but one of ordinary skill in the art mayrecognize that many further combinations and permutations of the subjectinnovation are possible. Accordingly, the disclosed subject matter isintended to embrace all such alterations, modifications, and variationsthat fall within the spirit and scope of the appended claims.

In particular and in regard to the various functions performed by theabove described components, devices, circuits, systems and the like, theterms (including a reference to a “means”) used to describe suchcomponents are intended to correspond, unless otherwise indicated, toany component which performs the specified function of the describedcomponent (e.g., a functional equivalent), even though not structurallyequivalent to the disclosed structure, which performs the function inthe herein illustrated exemplary aspects of the disclosed subjectmatter. In this regard, it will also be recognized that the disclosedsubject matter includes a system as well as a computer-readable mediumhaving computer-executable instructions for performing the acts and/orevents of the various methods of the disclosed subject matter.

In addition, while a particular feature of the disclosed subject mattermay have been disclosed with respect to only one of severalimplementations, such feature may be combined with one or more otherfeatures of the other implementations as may be desired and advantageousfor any given or particular application. Furthermore, to the extent thatthe terms “includes,” and “including” and variants thereof are used ineither the detailed description or the claims, these terms are intendedto be inclusive in a manner similar to the term “comprising.”

In this application, the word “exemplary” is used to mean serving as anexample, instance, or illustration. Any aspect or design describedherein as “exemplary” is not necessarily to be construed as preferred oradvantageous over other aspects or designs. Rather, use of the wordexemplary is intended to present concepts in a concrete fashion.

Various aspects or features described herein may be implemented as amethod, apparatus, or article of manufacture using standard programmingand/or engineering techniques. The term “article of manufacture” as usedherein is intended to encompass a computer program accessible from anycomputer-readable device, carrier, or media. For example, computerreadable media can include but are not limited to magnetic storagedevices (e.g., hard disk, floppy disk, magnetic strips . . . ), opticaldisks [e.g., compact disk (CD), digital versatile disk (DVD) . . . ],smart cards, and flash memory devices (e.g., card, stick, key drive . .. ).

What is claimed is:
 1. A node system, comprising: a memory that storesexecutable components; and a processor, operatively coupled to thememory, that executes the executable components, the executablecomponents comprising: a data input component configured to receiveindustrial data from an adjacent upstream node system of a data pipelineand to queue the industrial data to yield queued data; a data outputcomponent configured to send subsets of the queued data to an adjacentdownstream node system of the data pipeline in data batches; a batchsize calculation component configured to set a size of a next databatch, of the data batches, to be sent to the adjacent downstream nodesystem based on performance metric data received from the adjacentdownstream node system, wherein the batch size calculation component isconfigured to set the size of the next data batch by offsetting the sizeof the next data batch relative to a default data batch size, and theamount of the offsetting is based on a difference between a downstreamprocessing latency reported by the performance metric data and anexpected processing latency; and a batch size setting componentconfigured to instruct the data output component to send the next databatch in accordance with the size determined by the batch sizecalculation component.
 2. The node system of claim 1, wherein the batchsize calculation component is configured to reduce the size of the nextdata batch as a function of increasing downstream processing latency asdetermined based on the downstream processing latency reported byperformance metric data.
 3. The node system of claim 1, wherein theperformance metric data comprises processing latency information for theadjacent downstream node system and one or more other node systemslocated downstream relative to the adjacent downstream node system, andthe batch size calculation component is configured to determine, as thedownstream processing latency, a cumulative downstream processinglatency based on the processing latency information.
 4. The node systemof claim 3, wherein the batch size calculation component is configuredto determine the cumulative downstream processing latency as a sum of afirst processing latency of the adjacent downstream node system and atleast one second processing latency of the one or more other nodesystems located downstream relative to the adjacent downstream nodesystem, and the first processing latency and the second processinglatency are reported in the performance metric data.
 5. The node systemof claim 3, wherein the amount of the offsetting is further based on aproduct of the difference between the cumulative downstream processinglatency and the expected processing latency and a coefficientrepresenting a slope of a defined relationship between data batch sizeand processing latency.
 6. The node system of claim 1, wherein theperformance metric data is first performance metric data, and theexecutable components further comprise a performance monitoringcomponent configured to send second performance metric data to theadjacent upstream node system, the second performance metric datareporting a processing latency of the node system and of the adjacentdownstream node system.
 7. The node system of claim 1, furthercomprising a timestamp alignment component configured to control atiming of release of the next data batch based on a reference timestampreceived from a time moderator agent.
 8. The node system of claim 7,wherein the timestamp alignment component is configured to permitrelease of the next data batch during a current data transmission cycleof the data pipeline in response to determining that an oldest timestampassociated with the queued data is not more recent than the referencetimestamp in excess of a defined lag time.
 9. A method, comprising:queuing, by a node system comprising a processor, industrial datareceived from an adjacent upstream node system of a data pipeline toyield queued data; setting, by the node system, a size of a next databatch to be sent to an adjacent downstream node system of the datapipeline based on performance metric data received from the adjacentdownstream node system, wherein the setting comprises: determining anoffset amount based on a difference between a downstream processinglatency reported by the performance metric data and an expectedprocessing latency, and offsetting, by the offset amount, the size ofthe next data batch relative to a default data batch size; and sending,by the node system, a subset of the queued data to the adjacentdownstream node system of the data pipeline as the next data batch inaccordance with the size.
 10. The method of claim 9, wherein the settingof the size of the next batch comprises reducing the size of the nextdata batch as a function of increasing downstream processing latency asdetermined based on the performance metric data.
 11. The method of claim9, wherein the performance metric data comprises processing latencyinformation for the adjacent downstream node system and for one or moreother node systems located downstream relative to the adjacentdownstream node system, and the setting of the size of the next batchcomprises: determining, as the downstream processing latency, acumulative downstream processing latency based on the processing latencyinformation, and setting the size of the next data batch based on thecumulative downstream processing latency.
 12. The method of claim 11,wherein the determining of the cumulative downstream processing latencycomprises determining the cumulative downstream processing latency as asum of a first processing latency of the adjacent downstream node systemand at least one second processing latency of the one or more other nodesystems located downstream relative to the adjacent downstream nodesystem.
 13. The method of claim 11, wherein the setting of the offsetamount comprises determining the offset amount further based on aproduct of the difference between the cumulative downstream processinglatency and the expected processing latency and a coefficientrepresenting a slope of a defined relationship between data batch sizeand processing latency.
 14. The method of claim 9, wherein theperformance metric data is first performance metric data, and the methodfurther comprises sending, by the node system, second performance metricdata to the adjacent upstream node system, the second performance metricdata reporting a processing latency of the node system.
 15. The methodof claim 9, wherein the sending comprises controlling a release time ofthe next data batch based on a reference timestamp received from a timemoderator agent.
 16. The method of claim 15, wherein the controlling therelease time comprises permitting release of the next data batch duringa current data transmission cycle of the data pipeline in response todetermining that an oldest timestamp associated with the queued data isnot more recent than the reference timestamp in excess of a defined lagtime.
 17. A non-transitory computer-readable medium having storedthereon instructions that, in response to execution, cause a node systemcomprising a processor and acting as a node of a data pipeline toperform operations, the operations comprising: queuing industrial datareceived from an adjacent upstream node system of the data pipeline toyield queued data; setting a size of a next data batch to be sent to anadjacent downstream node system of the data pipeline based onperformance metric data received from the adjacent downstream nodesystem, wherein the setting comprises offsetting the size of the nextdata batch relative to a default data batch size by an offset amountdetermined based on a difference between a downstream processing latencyreported by the performance metric data and a defined expectedprocessing latency; and sending a subset of the queued data to theadjacent downstream node system of the data pipeline as the next databatch in accordance with the size.
 18. The non-transitorycomputer-readable medium of claim 17, wherein the setting of the size ofthe next batch comprises reducing the size of the next data batch as afunction of increasing downstream processing latency as determined basedon the performance metric data.
 19. The non-transitory computer-readablemedium of claim 17, wherein the performance metric data comprisesprocessing latency information for the adjacent downstream node systemand for one or more other node systems located downstream relative tothe adjacent downstream node system, and the setting of the size of thenext batch comprises: determining, as the downstream processing latency,a cumulative downstream processing latency based on the processinglatency information, and setting the size of the next data batch basedon the cumulative downstream processing latency.
 20. The non-transitorycomputer-readable medium of claim 17, wherein the sending comprisescontrolling a release time of the next data batch based on a referencetimestamp received from a time moderator agent.